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Commentary 


Scientists  Urge  DHS  to  Improve  Bioterrorism 
Risk  Assessment 


Gregory  S.  Parnell,  Luciana  L.  Borio,  Gerald  G.  Brown,  David  Banks,  and  Alyson  G.  Wilson 


In  2006,  the  Department  of  Homeland  Security  (DHS)  completed  its  first  Bioterrorism  Risk  Assessment  (BTRA),  in¬ 
tended  to  be  the  foundation  for  DHS’s  subsequent  biennial  risk  assessments  mandated  by  Homeland  Security  Presidential 
Directive  10  (HSPD-10).  At  the  request  of  DHS,  the  National  Research  Council  established  the  Committee  on  Method¬ 
ological  Improvements  to  the  Department  of  Homeland  Security’s  Biological  Agent  Risk  Analysis  to  provide  an  indepen¬ 
dent,  scientific  peer  review  of  the  BTRA.  The  Committee  found  a  number  of  shortcomings  in  the  BTRA,  including  a  fail¬ 
ure  to  consider  terrorists  as  intelligent  adversaries  in  their  models,  unnecessary  complexity  in  threat  and  consequence 
modeling  and  simulations,  and  a  lack  of  focus  on  risk  management.  The  Committee  unanimously  concluded  that  an  im¬ 
proved  BTRA  is  needed  to  provide  a  more  credible  foundation  for  risk-informed  decision  making. 

“The  threat  posed  by  biological  agents  employed  in  a  terrorist  attack  on  the  United  States  is  arguably  the  most  important 
homeland  security  challenge  of  our  era.  Whether  natural  pathogens  are  cultured  or  neiv  variants  are  bioengineered,  the 
consequence  of  a  terrorist-induced  pandemic  could  be  millions  of  casualties — far  more  than  we  woidd  expect  from  nuclear 
terrorism,  chemical  attacks,  or  conventional  attacks  on  the  infrastructure  of  the  United  States  such  as  the  attacks  of  Sep¬ 
tember  11,  2001.  Even  if  there  were  fewer  casualties,  additional  second-order  consequences  (including  psychological,  social, 
and  economic  effects)  would  dramatically  compound  the  effects.  Bioengineering  is  no  longer  the  exclusive  purview  of  state 
sponsors  of  terrorism;  this  technology  is  now  available  to  small  terrorist  groups  and  even  to  deranged  individuals.  ” 

— Department  of  Homeland  Security’s  Biological  Threat  Risk  Assessment: 

A  Call  for  Change,  National  Research  Council,  2008 


DHS’s  First  Biological  Threat 
Risk  Assessment 

Risk  assessment  (the  quantification  of  risk)  is  the  founda¬ 
tional  element  of  risk  analysis,  which  also  includes  risk 


communication  (the  provision  of  information  about  risks) 
and  risk  management  (strategies  for  reducing  future 
losses).1'3  Quantifying  risk  is  the  prerequisite  for  effective 
risk  communication  to  policymakers  and  stakeholders,  and 
for  supporting  critical  risk  management  decisions  by  all  lev- 
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els  of  government  and  the  private  sector.  Because  it  is  not 
possible  to  fully  protect  every  target  and  community  against 
every  potential  threat,  we  need  to  identify  the  greatest  risks 
and  take  the  most  efficient  steps  to  reduce  them. 

Homeland  Security  Presidential  Directive  10  (HSPD- 
10):  Biodefense  for  the  21st  Century,4  issued  in  April  2004, 
states  that  “  [b]  iological  weapons  in  the  possession  of  hostile 
states  or  terrorists  pose  unique  and  grave  threats  to  the 
safety  and  security  of  the  United  States  and  our  allies”  and 
charges  the  Department  of  Homeland  Security  (DHS)  with 
issuing  biennial  assessments  of  biological  threats,  to  “guide 
prioritization  of  our  on-going  investments  in  biodefense-re- 
lated  research,  development,  planning,  and  preparedness.”4 
The  subsequent  Homeland  Security  Presidential  Directive 
18  (HSPD-18):  Medical  Countermeasures  against 

Weapons  of  Mass  Destruction5  calls  for  an  integrated  risk 
assessment  of  all  chemical,  biological,  radiological,  and  nu¬ 
clear  (CBRN)  threats. 

In  2006,  DHS  completed  its  first  risk  assessment  and 
published  Bioterrorism  Risk  Assessment  (BTRA).6  The  work 
was  contracted  to  Battelle  Memorial  Institute  in  Colum¬ 
bus,  Ohio.  The  BTRA  report  resulted  from  a  complex  fed¬ 
eration  of  integrated  computer-based  models  to  estimate 
the  risks  associated  with  the  intentional  terrorist  release  of 
each  of  27  natural  pathogens7  and  one  engineered  agent 
(multidrug  resistant  Bacillus  anthracis).  The  BTRA  ranks 
each  pathogen  according  to  its  level  of  risk,  based  on  sub¬ 
jective  event  probabilities  and  their  consequences.  The  sub¬ 
jective  event  probabilities  were  elicited  from  dozens  of 
biological  weapons  experts.  Event  consequences  were  esti¬ 
mated  from  a  number  of  different  models  and  simulations. 
For  example,  to  estimate  the  spread  and  health  effects  of  in¬ 
fectious  diseases,  “susceptible,  exposed,  infected,  and  re¬ 
moved”  (SEIR)  mathematical  models  for  the  relevant  dis¬ 
eases  were  employed  (see  Bailey8  for  a  comprehensive 
treatment). 

DHS  promotes  the  BTRA  of 2006  as  an  “end-to-end  risk 
assessment  of  the  bio  terrorism  threat,”  ranking  various 
agents  by  their  level  of  risk,  to  “assist  and  guide  biodefense 
strategic  planning.”6  The  intent  of  BTRA  is  that  it  could  be 
used  to  identify  vulnerabilities  to  particular  agents  or  sce¬ 
narios,  or  to  prioritize  federal  investment  in  particular 
countermeasures.  Moreover,  the  BTRA  methodology  is  in¬ 
tended  to  be  the  foundation  for  future  biennial  risk  assess¬ 
ments  by  DHS  and  perhaps  to  be  applied  to  other  areas 
such  as  chemical  or  nuclear  terrorism,  fulfilling  DHS’s 
obligations  to  HSPD-10  and  HSPD-18. 

NRC  Committee  Finds  Serious 
Limitations  with  DHS’s  BTRA 


At  the  request  of  DHS,  the  National  Research  Council  es¬ 
tablished  the  Committee  on  Methodological  Improve¬ 


ments  to  the  Department  of  Homeland  Security’s  Biologi¬ 
cal  Agent  Risk  Analysis  to  provide  an  independent,  scien¬ 
tific  peer  review  of  the  BTRA.  The  Committee  reviewed  all 
of  the  details  in  the  BTRA  of  2006,  interviewed  its  leaders 
and  its  implementers,  held  discussions  with  other  experts 
who  had  not  themselves  participated  in  the  BTRA,  and  re¬ 
ceived  briefings  from  DHS  on  planned  improvements  to 
the  subsequent  BTRA  of  2008.  In  accord  with  HSPD-10, 
the  Committee  reviewed  not  only  the  mathematical  foun¬ 
dation  of  the  BTRA,  but  also  its  potential  utility  to  policy¬ 
makers. 

The  NRC  Committee’s  review  was  limited  to  the  initial 
BTRA  of  2006  because  the  Committee’s  deliberations 
ended  before  the  BTRA  of  2008  report  was  released.  Al¬ 
though  the  updated  BTRA  of  2008  included  a  number  of 
additions  and  refinements  to  the  BTRA  of  2006,  none  of 
the  changes  would  alter  the  Committee’s  assessment  of  the 
BTRA  methodology  or  render  the  Committee’s  findings  ir¬ 
relevant. 

The  Committee  identified  a  number  of  fundamental 
problems  with  the  BTRA,  ranging  from  the  use  of  unneces¬ 
sarily  complicated  probability  models,  to  simplistic  as¬ 
sumptions  regarding  the  manner  in  which  terrorist  behav¬ 
ior  should  be  modeled.9  The  Committee  also  unanimously 
judged  the  BTRA  unsuitable  for  risk  management.  In  their 
final  report,  the  Committee  made  detailed  recommenda¬ 
tions  as  to  ways  DHS  could  remediate  the  BTRA’s  many 
shortcomings. 

DHS  Failed  to  Adequately  Model 
the  Behavior  of  Terrorists  as 
Intelligent  Adversaries 

Fortunately,  bio  terrorism  attacks  have  been  so  infrequent 
that  we  have  limited  data  on  which  to  draw.  However,  as 
9/11  demonstrated,  terrorists  will  certainly  design  their  at¬ 
tacks  to  exploit  our  vulnerabilities  and  attempt  to  achieve 
consequences  that  will  meet  their  objectives.  Therefore, 
homeland  security  risk  analysts  must  consider  the  potential 
decisions  of  terrorists  as  if  they  were  intelligent  adversaries. 

However,  the  BTRA  does  not  consider  the  range  of  pos¬ 
sible  attack  strategies  that  an  intelligent  adversary  might 
pursue.  Instead,  the  BTRA  uses  an  event  tree,  representing 
many  hypothetical  sequences  of  events,  from  the  terrorist 
decision  to  initiate  an  attack  to  the  consequences  of  the  at¬ 
tack.  It  represents  terrorist  decisions  by  means  of  probabili¬ 
ties  assessed  by  experts  as  they  would  assess  the  probability 
of  a  natural  hazard  (eg,  an  earthquake)  or  an  engineered 
system  failure  (eg,  a  nuclear  reactor).  For  example,  in  the 
BTRA  experts  assign  subjective  probabilities  for  terrorists 
selecting  a  target,  selecting  a  pathogen,  acquiring  the  patho¬ 
gen,  etc.  Each  of  these  events  is  associated  with  a  very  small 
probability.  To  assess  the  expected  consequences  of  each 
scenario,  the  assigned  small  probability  of  each  event  must 
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be  multiplied  for  every  event  in  the  scenario.  This  yields  a 
small  number,  minimizing  the  true  consequences  of  each 
threat. 

An  analogy  to  the  events  of  9/ 1 1  might  be  useful  here.  If 
the  BTRA  were  to  be  used  to  characterize  the  threat  of 
9/11,  experts  would  have  assigned  probabilities  for  Al- 
Qaeda  to  choose  the  World  Trade  Center  towers  as  the  tar¬ 
get  and  planes  as  the  weapon.  These  small  probabilities, 
when  multiplied  by  the  number  of  deaths  caused  by  such 
an  attack,  would  have  resulted  in  a  very  small  number,  and 
the  risk  of  such  attack  would  have  been  deemed  insignifi¬ 
cant.  However,  terrorists,  our  intelligent  adversaries,  did 
not  assign  probabilities  to  each  of  their  choices.  Terrorists 
are  goal-oriented,  resourceful  adversaries,  who  will,  given 
the  constraints  they  perceive,  select  the  best  agent  and  tar¬ 
get  to  achieve  their  objectives. 

The  Complexity  of  the  DHS 
Consequence  Models  Is  Not  Supported 
by  Existing  Knowledge 

The  Committee  closely  examined  the  assumptions  and  the 
mathematical  details  of  the  BTRA  and  found  weaknesses  in 
model  conception  and  unnecessary  complexity  throughout, 
along  with  errors  in  enough  of  the  underlying  mathematics 
and  statistics  that  the  Committee  was  compelled  to  express 
significant  concern. 

Examples  of  unnecessary  complexity  include  the  large 
number  of  events  in  each  tree  and  the  overly  elaborate  algo¬ 
rithm  with  which  outcome  probabilities  are  calculated.  Ad¬ 
ditionally,  the  SEIR  models  used  to  analyze  the  health  con¬ 
sequences  of  various  attack  scenarios  require  input 
parameter  values  for  which  empirical  data  do  not  exist.  Be¬ 
cause  the  granularity  of  detail  in  the  SEIR  models  cannot 
be  supported  by  existing  clinical  and  epidemiologic  data  for 
any  pathogen  on  the  BTRA  list — not  even  B.  anthracis — 
many  broad  assumptions  are  required.  This  results  in  an  il¬ 
lusion  regarding  the  precision  of  the  results  when  there  is 
none. 

BTRA  Does  Not  Focus  on 
Risk  Management 

Risk  assessment  requires  an  estimate  of  the  threat  (based  on 
the  intentions  and  capabilities  of  adversaries)  and  the  ex¬ 
pected  magnitude  of  consequences  (eg,  deaths,  illnesses, 
and  economic  losses)  given  our  vulnerabilities.  Alone,  com¬ 
plex  biennial  risk  assessments,  such  as  the  BTRA,  have  no 
direct  impact  on  risk  reduction.  In  addition,  the  risk  has  to 
be  communicated  to  the  stakeholders  and  policymakers.  In 
the  end,  only  effective  risk  management  strategies  can  re¬ 
duce  risk. 

However,  the  current  versions  of  the  BTRA  present  sig¬ 
nificant  barriers  to  stakeholders  who  should  be  using  the 


BTRA  for  risk  management.  The  tool’s  construction  and 
assumptions  are  not  transparent,  the  interface  is  not  user- 
friendly,  and  the  tool  does  not  easily  allow  modeling  of  al¬ 
ternative  scenarios  and  evaluation  of  various  risk  manage¬ 
ment  strategies.  Given  the  large  number  of  homeland 
security  stakeholders  who  influence  the  vulnerabilities  and, 
as  a  result,  the  consequences  of  a  biological  attack,  it  is  crit¬ 
ically  important  that  DHS  revise  its  tool  so  that  these  stake¬ 
holders  can  assess  the  potential  effects  of  their  risk  manage¬ 
ment  strategies. 

The  Call  for  Change 


The  threat  of  bioterrorism  is  significant;  various  pathogens 
could  cause  unprecedented  harm  to  the  health  and  prosper¬ 
ity  of  Americans,  and  the  technical  barriers  to  launching  ef¬ 
fective  biological  attacks  are  not  insuperable,  even  for  small 
groups  or  individuals.  Thus,  the  objectives  of  the  BTRA  are 
essential  and  laudable.  But  given  the  defects  in  the  BTRA’s 
underlying  model  and  user  interface,  the  Committee  con¬ 
cluded  that  the  current  incarnation  of  the  BTRA  is  inade¬ 
quate,  does  not  satisfy  the  intent  of  HSPD-10,  and  should 
not  be  used  as  the  foundation  for  future  biennial  risk  assess¬ 
ments  or  expanded  to  an  integrated  risk  assessment  of 
CBRN  threats.  The  Committee  unanimously  believes  that 
an  improved  BTRA  is  needed  to  provide  a  credible  founda¬ 
tion  for  risk-informed  decision  making. 

What  DHS  Should  Do 


A  flawed  probabilistic  risk  assessment  tool  is  inadequate  to 
communicate  risk  and  to  support  risk  management  deci¬ 
sions.  The  risk  of  a  bioterrorism  attack  requires  the  best  ef¬ 
forts  of  our  homeland  security  community.  The  authors  be¬ 
lieve  that  DHS  should  take  3  near-term  actions  to  make  the 
necessary  changes  to  BTRA: 

1.  Form  an  independent  senior  technical  review  panel  to 
advise  the  DHS  risk  analysis  team  leaders.  The  DHS 
risk  assessment  is  a  critical  challenge  that  requires  exper¬ 
tise  in  many  technical  disciplines  (including  risk  analy¬ 
sis,  biology,  epidemiology,  statistics,  operations  re¬ 
search,  social  science,  and  economics)  to  develop 
credible  and  responsive  risk  assessment  models.  The  se¬ 
nior  review  panel  should  guide  the  risk  analysis  team  in 
the  implementation  of  the  required  changes  described  in 
the  NRC  report  and  review  the  BTRA  output.  Further, 
this  review  panel  should  communicate  to  government 
leaders  and  stakeholders  DHS’s  progress  in  revising  the 
BTRA. 

2.  Obtain  robust  stakeholder  input  and  feedback.  DHS 

must  meaningfully  engage  with  stakeholders  and  incor- 
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porate  their  feedback  into  the  BTRA.  Many  in  govern¬ 
ment  and  public  and  private  organizations  need  to  par¬ 
ticipate  in  the  bioterrorism  risk  assessment  to  under¬ 
stand  their  vulnerabilities,  the  potential  consequences  of 
attack,  and  the  effectiveness  of  mitigation  strategies. 
The  results  of  the  assessments  should  directly  inform 
their  risk  management  decisions. 

3.  Develop  and  incorporate  risk  analysis  techniques  that 
model  terrorists  as  intelligent  adversaries.  The  NRC 
report  proposed  3  possible  techniques  to  assess  the  im¬ 
pact  of  an  intelligent  adversary.  What  these  techniques 
share  is  that  instead  of  requiring  the  analyst  to  input 
probabilities  of  what  the  terrorist  is  likely  to  do,  they  in¬ 
stead  output  the  terrorist  actions  that  will  best  achieve 
the  terrorists’  objectives.  The  3  techniques  include:  (1)  a 
“bioterrorism  decision  model”  to  model  terrorist  actions 
and  U.S.  strategic  actions  as  decisions,  and  agent  acqui¬ 
sition,  employment,  vulnerabilities,  consequences,  and 
mitigation  events  as  uncertain  events  using  available  off- 
the-shelf  software  (eg,  see  Decision  Analysis  Software 
Survey,  http://Lionhrtpub.com/orms/surveys/das/das. 
html);  (2)  a  tri-level  decision  support  model  to  allocate 
defensive  investments  (visible  to  the  attacker)  that  repre¬ 
sent  an  attacker’s  reasonable  response  to  observing  these 
preparations,  and  reactions  to  any  attack  with  the  re¬ 
sources  made  available  by  the  defensive  investments; 
and  (3)  a  game-theoretic  model  of  the  adversaries  that 
randomizes  expected  consequences  to  capture  the  vari¬ 
ability  of  outcomes.  These  are  not  mere  theoretical  sug¬ 
gestions,  but  rather  substantive  tools  and  methods 
drawn  from  extensive  research  and  experience  in  the 
military  and  in  the  private  sector  that  can  significantly 
improve  the  credibility  and  usefulness  of  the  BTRA.  We 
recommend  that  these  3  techniques  (or  others  that 
achieve  the  same  objectives)  be  developed  and  evaluated 
as  soon  as  possible.  Insights  from  multiple  credible 
models  may  be  the  best  way  to  ensure  that  homeland  se¬ 
curity  decision  makers  have  the  best  available  data  for 
informed  decision  making. 

In  their  report  to  the  National  Research  Council,  the 
Committee  has  made  further  detailed  and  technical  recom¬ 
mendations  that  would  improve  the  BTRA.1 2 3 4 5 6 7 8 9  The  Commit¬ 
tee  hopes  DHS  will  now  complete  the  steps  necessary  to 
produce  a  robust  and  reliable  risk  assessment  to  meet  this 
urgent  need. 
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